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FOREWORD 

Tliis  research  was  performed  under  Exploratory  Development  Task  Area  PF.55.522.102 
(Technique  for  Evaluating  Marine  Corps  Training  in  Tactical  Field  Exercises).  The  researcl.  was  initiated 
in  1 972  in  response  to  a request  from  the  Marine  Corps  Development  and  Education  Command  to  develop 
unit  performance  measures  and  criteria  for  use  in  tactical  field  exercises  in  conjunction  with  tlie  Tactical 
Warfare  Analysis  and  Evaluation  System.  Prior  research  in  this  areaEas  bcen  described  in  NPRDC  TR  74-1 1 , 
Survey  of  Unit  Performance  Effectiveness  Measures  by  OrvmAT  Larson,  Stephen  I.  Sander,  aii3  JoluTTTr 
Stcinemann,  January  1974. 


J.  J.CLARKIN 
Commanding  Officer 


SUMMARY 
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OBJECTIVE 

Tiic  primary  objective  of  this  research  effort  was  to  develop  unit  effectiveness  measures  and  criteria 
for  use  in  evaluating  Marine  Corps  field  exercises. 


BACKGROUND 

This  'eport  documents  the  second  phase  of  an  ongoing  research  effort  in  support  of  the  Marine  Corps 
Tactical  Warfare  Analysis  and  Evaluation  System  (TWAES),  a computer-based  system  for  use  in  field  exercise 
environments.  The  previous  phase  was  a review  of  the  state  of  the  art  in  performance  evaluation  systems  and 
was  documented  in  NPRDC  TR  74-1 1 (January  1974).  The  primary  existing  sources  of  data  on  unit  per- 
formance are  checklists  and  the  subjective  experience  of  senior  Marine  Corps  officers.  An  evaluation  format 
which  could  account  for  the  effects  of  contextual  variables  which  moderate  performance  was  deemed  nec- 
essary. This  led  to  a requirement  for  the  development  of  contextual  items  and  performance  response  items 
for  use  in  an  evaluation  system. 


METHOD 

The  DELPHI  — a technique  for  systematically  extracting.expert  opinion  — was  selected  and  utilized 
as  the;primary  research  approach  to  objectify  the  evaluation  policy  of  senior  Marine  Corps  officers.  Infantry 
battalion  commanders^were  selected  to  participate  as  subject  matter  experts.  The  DELPHI  materials  were 
distributed  to  participants  spread  over  a wide  geographiaarea,  and  returned  by  them  via  the  mail. 


RESULTS 

The  use  of  the  DELPHI  resulted'in  the  development  of  L7  contextual  variables.  These  were  amenable 
to  categorization  in  terms  of  situational  and  physical  variables.  Response  items  of  two  types  were  developed: 
(1)  observable  performance  items  and  (2)  performance  items  which  are  usually  less  than  fully  observable 
during  an  exercise  but  are  nonetheless  considered  important.  These  items  were  amenable  to  breakdown  into 
command,  unit,  and  individual  performance.  Data  were  also  developed  on  the  average  times  needed  to  eval- 
uate units  of  varying  size  ranging  from  squad  to  brigade  in  field  exercise  environments. 


CONCLUSIONS  AND  RECOMMENDATIONS 

The  DELPHI  technique  is  effective  for  extracting  policy  information  from  a widely  dispersed  group 
of  participants.  Minor  changes  made  in  the  classical  DELPHI  format  to  adapt  it  to  this  application  were 
foundsto  be  nonsignificant.  Performance  evaluation  items  relating  the  performance  response  to  the  context 
of  thaLresponse  and  expected  evaluation  times  were  developed.  It  is  recommended  that  the  performance 
contextual  and=response  items  developed  by  this  DELPHI  research  be  integrated  into  the  exercise  evaluation 
format,  and  that  initial  reliability  and  validity  data  be  gathered  from  the  field. 
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OBJECTIVES 

This  researcli  was  conducted  In  support  of  the  Marine  Corps  Tactical  Warfare  Analyses  and  Evalua- 
tion System  .TWAES)  requirements.  The  primary  objectives  of  the  effort  were  to  develop  unit  effectiveness 
measures  and  criteria  for  use  in  performance  evaluation.  More  specifically,  the  objective  of  this  phase  of  the 
research  was  to  develop  a scheme  for  evaluating  unit  performance  as  moderated  by  contextual  or  environ- 
mental factors  such  as  mission,  terrain,  weather,  and  tactical  situation. 


BACKGROUND 


DOCUMENTATION 

This  report  documents  the  second  phase  of  a research  effort  in  support  of  the  Marine  Corps  TWAES 
reouirements.  The  TWAES  is  a computer-based  system  designed  to  improve  the  control  and  evaluative 
aspects  of  Marine  Corps  field  exercises  and  thereby  to  increase  the  value  and  effectiveness  of  the  training 
exercises.  The  initial  phase  of  the  research  consisted  of  a review  of  the  state  of  the  art  of  existing  evaluation 
systems  and  of  the  performance  evaluation  literature.  That  effort  was  documented  in  NPRDC  TR  74-1 1 
of  January  1974,  (Larson,  Sander,  and  Steinemann,  1974)  which  indicated  that  improved  performance  eval- 
uation techniques  and  criteria  were  needed  to  complement  and  fully  utilize  the  advanced  capabilities  offered 
by  the  TWAES  concept.  It  also  cited  the  importance  of  contextual  factors  in  performance  evaluation,  and 
discussed  the  potential  applicability  of  DELPHI  procedures  to  TWAES  research  requirements. 


CRITERIA  DEVELOPMENT 

Organizations  which  exist,  either  by  necessity  or  design,  in  a competitive  arena  normally  evolve 
internal  functions  which  serve  to  ensure  theii  continued  competitiveness  and,  thereby,  their  survival.  Mili- 
tary organizations  arc  no  exception,  and  their  past  performance  record  is  one  measure  of  the  adaptiveness  of 
their  internal  evaluative  processes.  The  long-term  record  of  the  Marine  Corps  indicates  that  its  internal  eval- 
uative procedures  and  criteria  have  been  adequate  to  ensure  its  continued  competitiveness;  and  that  therein 
lies  a base  of  evaluative  expertise.  In  part,  this  expertise  has  been  codified  in  the  form  of  various  training 
evaluation  documents  and  publications.  An  additional  portion  of  the  evaluative  expertise,  however,  is  the 
.subjective  expertise  ofsenior  officers,  which  is  developed  over  an  extended  period  of  service  experience  and 
varies  from  individual  to  individual.  To  the  extent  that  this  expertise  is  codified,  it  is  readily  usable  for  eval- 
uative purpo.ses.  There  are  difficulties,  however,  in  accessing  and  utilizing  noncodified  expertise. 

Traditionally,  group  processes,  such  as  holding  conferences  or  convening  committees,  have  been  em- 
ployed to  extract  information  involving  subjective  data.  These  group  processes,  however,  provide,  at  best, 
opinions  subject  to  .such  drawbacks  as  the  bandwagon  effect,  specious  persuasion,  and  domineering  person- 
alities. Additionally,  there  is  the  possibility  that  the  final  decision  or  position  is  formulated  largely  by  the 
chairman  or  group  leader,  and  does  not  accurately  represent  the  combined  contributions  of  the  individual 
committee  participants.  In  essence,  conventional  group  processes  often  tend  to  stifle  the  contributions  of 
participants  through  goal-oriented  conformity.  In  an  attempt  to  overcome  these  and  related  procedural 
problems,  Helmer  and  Dalkey  of  the  Rand  Corporation  developed  the  DELPHI  technique  in  the  1950’s 
(Dalkey  and  Helmer,  1963). 

The  DELPHI  may  be  used  as  aipolicy-capture  technique  by  which  the  internalized  policy  consensus  of 
a group  of  experts  are  subjected  to  a series  of  extensive  interrogations  interspersed  with  controlled  feedback 
from  other  experts.  This  technique  ensures  that  all  positions  on  an  issue  have  been  puf  forward  for  considera- 
tion, and  examines  the  acceptability  of  any  particular  position  (Turoff,  1970).  This  procedure  encourages 
participants  to  consider  factors  they  may  have  overlooked  or  initially  considered  irrelevant  to  the  issue  at 
hand.  This  approach  of  systematically  extracting  expert  opinion,  when  complemented  by  existing  sources  of 
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vodilied  cx.iki.itioii  materials,  is  relevant  to  TWAES  requirements  lor  the  development  of  unit  perfurmaiKc 
«.riteria.  These  criteria  would  be  used  in  evaluating  unit  performance  in  field  exercise  environments. 

EVALUATIVE  FORMAT 

TWAES  training  and  evaluation  requirements  involve  the  recording  and  evaluation  of  unit  and  indi- 
vidual performance  in  field  exercise  environments.  This  performance  evaluation  requires  that  contextual 
f.ictors  which  moderate  performance  be  considered.  In  other  words,  the  context  in  which  the  performance 
is  required  must  be  carefully  specified,  since  environmental  factors  can  significantly  affect  the  range  and 
quality  of  expected  resixjnses  in  any  given  situation.  Currently,  the  Marine  Corps  uses  trained  umpire  staffs 
to  input  evaluative  information  to  TWAES.  However,  the  subjectivity  of  the  assessment  variables  makes  it 
difficult  to  ensure  interumpire  agreement.  This  ix)tential  interrater  variability  is,  in  part,  controlled  by  the 
use  of  umpire  schools  and  the  assignment  of  umpires  to  evaluate  within  their  own  areas  of  specialization. 
This  is  particularly  important  when  units  arc  implicitly,  if  not  explicitly,  evaluated  against  each  other  as  well 
asfigainst  absolute  performance  levels.  An  evaluative  system  which  can  reduce  this  interratcr  variability, 
provide  improved  criteria,  and  normalize  ratings  across  units  to  account  for  contextual  variables  is  necessary. 

A sy.stem  that  allows  the  performance  of  all  units  to  be  evaluated  on  a common  or  normalized  scale, 
even  though  these  units  are  required  to  perform  in  dissimilar  environments,  is  most  desirable.  In  an  ideal 
sense,  this  capability  is  probably  not  within  the  current  state  of  the  art.  It  is,  however,  within  the  realm  of 
practical  approximation.  A similar  effort  was  conducted  for  the  Air  Force  Semiautomatic  Ground  Environ- 
ment (SAGE)  system  (Cunningham,  Shelden,  and  Zagorski,  1965  and  Parsons,  1972)  to  determine  the  ef- 
fects of  situational  variables  on  performance.  A portion  of  that  effort  (Project  NORM)  was  designed  for  the 
express  purpose  of  deriving  “improved  SAGE  performance  measures  and  the  development  of  normative 
rales  to  assess  changes  in  crew  performance  and  differences  between  crews”  (Sheldon  and  Zagorski,  1965). 
Tt.._  was  done  by  developing  a scoring  procedure  based  on  relative  scaling  which  was  “independent  of  the 
difficult,  of  a particular  mission  and  of  inalterable  sector  characteristics.  Crews  could  be  compared  even 
though  thtj  ■’ceived  different  mission  inputs  and  didmot  face  equivalent  environmental  circumstances” 
(Sheldon  and  Zagorski,  1965).  Such  a system  provides  normalized  outputs  for  the  comparison  of  units 
across  tasks  missions,  while  also  allowing  units  to  be  compared  against  either  absolute  criteria  or  their 
own  previous  performance. 


DEVELOPMENT  SEQUENCE 

The  development  of  a context-response  performance  evaluation  technique  for  TWAES  first  requires 
the  following: 

1.  The  identification  of  those  response  measures  which  accurately  reflect  unit  effectiveness. 

2.  The  identification  of  those  contextual  variables  which  moderate  unit  performance. 

3.  The  presence  of  reliable  observers  to  inpuLthese  contextual  and  response  factors. 

The  basic  information  for  both  the  response  and  contextual  factors  was  amenable  to  development  by  using 
the  DELPHI  technique  to  systematically  extract  and  objectify  the  existing  evaluative  policies  of  senior  field 
grade  officers.  This  information,  when  complemented  by  data  from  existing  literature  sources  on  perform- 
ance criteria  and  analyzed  statistically  for  interactions  and  nonlinearities,  would  comprise  the  initial  per- 
formance evaluation  system.  When  properly  formulated,  these  materials  could  be  entered  through  the 
existing  Digital  Message  Entry  Device  (DMED)  system  used  by  umpires,  and  would  allow  evaluative  data  on 
units  and  selected  individuals  to  be  accumulated  over  the  period  of  an  exericse.  Additionally,  a portion  of 
these  data  would  be  suitable  for  use  in  the  firepower  calculations  in  much  the  same  manner  as  the  existing 
tactical  effectiveness  factors  are  now  incorporated. 
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METHOD 


GENERAL 

The  DELPHI  tccliniquc  was  used  as  tlic  primary  research  procedure  in  the  development  of  the  con- 
textual and  response  variables.  Senior  Marine  Corps  officers  participated  as  evaluative  experts  in  the 
DELPHI.  This  approach  not  only  utilized  their  ex|>erience  but  also  provided  them  with  a participatory  role 
which  will  enhance  the  validity  of  the  criteria,  thereby  increasing  the  acceptability  of  theTWAES  evaluative 
procedures. 


PARTICIPANTS 

The  participants  selected  to  serve  as  experts  in  the  DELPHI  were  the  25  infantry  battalion  com- 
manders currently  serving  in  the  Marine  Corps.  As  a group,  these  officers  are  recognized  as  experts  within 
their  field.  Their  average  length  of  experience  was  22  years.  Since  ail  of  the  infantry  battalion  commanders 
were  contacted,  they  initially  comprised  a population  of  this  body  of  experts  rather  than  a sample.  To  en- 
courage willing  and  frank  participation,  they  were  contacted  directly  and  individually  asked  to  participate 
rather  than  indirectly  through  their  respective  command  chain.  Each  participant  received  a letter  inviting 
him  to  participate,  along  with  materials  on  the  goals  and  scope  of  the  project,  ivrucipants  were  asked,  but 
not  required,  to  provide  their  name  and  address  (and  forwarding  address,  if  appropriate)  so  that  contact 
could  be  maintained  for  the  duration  of  the  DELPHI.  Although  the  participants  were  asked  to  refrain  from 
discussing  the  issues  with  each  other,  they  could  use  any  other  informational  sources. 


PROCEDURE 

The  participants  received  and  returned  their  DELPHI  materials  by  mail.  They  were  asked  to  respond 
within  5 to  7 days.  Telephone  numbers  of  the  research  project  personnel  were  provided  to  participants  to 
facilitate  communications  in  the  event  questions  arose.  They  were  given  the  option  of  not  responding  to 
sections  of  the  matcrials;if  they  so  chose.  The  response  materials  returned  from  participants  in  each  round 
were  summarized,  statistically  analyzed,  and  reformatted  to  serve  as  feedback  to  participants  in  the  succeed- 
ing round.  Each  round  required  approximately  2 months  to  complete,  including  the  time  required  for  cor- 
respondence and  for  analyzing  the  materials.  A round-by-round  procedural  description  is  described  below. 


MATERIALS 

ROUND  ONE 

DELPHI  materials  were  sent  to  all  25  infantry  battalion  commanders.  Each  received  information  on 
the  background  of  the  project,  participation  instructions,  the  three  questions  listed  below,  and  preaddressed 
franked  envelopes  for  returning  the  materials.  (Project  and  general  information  distributed  to  participants 
are  contained  in  Appendix  A). 

1.  Question  1 

"WHEN  YOU  ARE  ASSIGNED  TO  EVALUATE  A UNIT  IN  A FIELD  EXERCISE  ENVIRON- 
MENT, WHAT  FACTORS  OR  CHARACTERISTICS  DO  YOU  USE  IN  FORMING  YOUR 
EVALUATION  OF  THE  UNIT’S  COMBAT  READINESS?" 
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2.  Question  2 

YOU  ARE  ASSIGNED  TO  EVALUATE  A UNIT  IN  A FIELD  EXERCISE  ENVIRON- 
MENT, HOW  LONG  DOES  IT  TAKE  YOU  TO  REACH  YOUR  DECISION  CONCERNING 
THE  UNIT'S  COMBAT  READINESS?  FOR  EXAMPLE,  DO  YOU  NORMALLY  USE  ALL  OF 
THE  TIME  A VAILABLE  IN  THE  EXERCISE  OR  DO  YOU  MAKE  YOUR  DECISION  SOME- 
TIME DURING  THE  EXERCISE  AND  USE  THE  REST  OF  THE  EXERCISE  PRIM  ARIL  Y TO 
CONFIRM  YOUR  DECISION?" 

3.  Question  3 

"WHEN  YOU  ARE  ASSIGNED  TO  EVALUATE  A UNIT  IN  A FIELD  EXERCISE  ENVIRON- 
MENT, WHA  T INFORMA TION  OR  FA CTORS  .ARE  USUALL  Y STILL  INCOMPLETE  A T 
THE  TIME  YOU  MAKE  OR  MUST  MAKE  YOUR  DECISION  CONCERNING  THE  UNIT'S 
COMBAT  READINESS?" 


ROUND  TWO 

Materials  for  this  round  were  sent  by  “name”  to  those  commanders  who  had  supplied  such  informa- 
tion on  the  previous  round,  and  by  “unit”  to  those  who  either  Iiad  not  responded  in  tlie  first  round  or  had 
elected  to  do  so  anonymously.  The  materials  submitted  in  the  first  round  were  compiled  and  formatted  to 
serve  as  feedback  for  the  second  round.  The  responses  submitted  to  Questions  1 and  3 were  reformatted  to 
facilitate  scaling.  General  information  reflecting  the  purposes  of  the  second  round  was  also  provided  (sec 
Appendix  A). 

1 . Question  1 : 

Participants  were  given  a list  of  48  items  submitted  in  response  to  Question  1 during  Round  1 (see 
Table  1 and  Appendix  B),  and  asked  to  rate  these  items  on rating  scale  (see  Figure  1 ).  Instructions  pro- 
vided and  the  original  question  are  shown  below. 

a.  Instructions: 

“The  following  items  were  submitted  in  response  to  Question  1 during  Round  One  of  the 
DELPHI.  You  are  now  asked  to  evaluate  each  of  the  items  listed  as  to  its  importance  in  accu- 
rately evaluating  a unit,  the  unit  level(s)  at  which  each  is  most  applicable,  and  how  often  or 
how  frequently  the  item  occurs  or  is  observable  in  a:  field  exercise.  Rating  scales  are  provided 
along  with  a restatement  of  the  original  question.  Circle  or  check  the  numbei  on  the  impor- 
tance and  frequency  rating  scales  and  fill  in  the  appropriate  number  or  numbers  in  the  unit  size 
blank.  Note;  Given  the  differences  between  combat  and  field  exercises  environments,  the  im- 
portance scale  as  used  here  refers  to  the  importance  of  the  item  or  factor  as  an  evaluative  item 
and  as  an  indicator  of  combat  readiness.  An  additional  page  is  provided  following  the  it*  p’s  to 
allow  you  to  rephrase  items  and  to  add  additional  items.” 

b.  Question: 

"WHEN  YOU  ARE  ASSIGNED  TO  EVALUATE  A UNIT  IN  A FIELD  EXERCISE 
ENVIRONMENT.  WHA  T FA  CTORS  OR  CHARA  CTERISTICS  DO  YOU  USE  IN  FORMING 
YOUR  EVALUATION  OF  THE  UNIT'S  COMBAT  READINESS?" 

2,  Question  2: 

Because  of  the  wide  range  of  responses  and  comments  received  in  response  to  Question  2 during 
Round  One,  the  question  was  reformatted  for  Round  2.  Participants  were  provided  with  a complete  listing 
of  comments  (not  included  in  this  report),  and  asked  to  respond  to  the  revised  question.  Instructions  pro- 
vided and  a restatement  of  the  original  question  are  shown  below. 
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TABLE  1.  SUMMARY  OF  DELPHI  ITEMS  DEVELOPED  FROM  QUESTION  1 . 


1.  Coordination 

2.  Supporting  arms 

3.  Noise  and  light,  discipline 

4.  Casualty  evacuation 

5.  Firc/maneuver 

6.  Analyze  mission/terrain 

7.  Discipline  command  post 

8.  Staff  coordination 

9.  Clear  and  timely  orders 

1 0.  Command  pacing 

1 1 . Communications 

1 2.  Encryption  and  shackling 

13.  Control  of  forces 

1 4.  Effectiveness  of  control 

15.  Special  situations 

1 6.  Logistics  support 

17.  Sustain  operationally 

18.  Defensive  principles 

19.  Organizational  equipment 

20.  Individual  equipment 

21.  Fire  discipline 

22.  Timeliness  of  fire  support 

23.  Information  flow 

24.  Intelligence  gathering 

25.  Leader’s  professional  skill 


26.  MOS  skills 

27.  Mission  accomplishment 

28.  MOS  examinations 

29.  Ability  to  navigate 

30.  Offensive  capability 

31 . Patrolling  capability 

32.  Replacement  status 

33.  Physical  condition 

34.  Moralc/motivation 

35.  Esprit  and  cohesiveness 

36.  Appearance  of  troops 

37.  Develop  plans  and  orders 

38.  Tactical  principles 

39.  Protective  measures 

40.  Change  of  billet 

41.  Supply  economy 

42.  Reporting  procedures 

43.  Reaction  to  themnexpected 

44.  Command  flexibility 

45.  Team  functions 

46.  Crew-served  weapons 

47.  Discipline 

48.  Innovative  SOPs 

49.  Special  equipment 

50.  Secure  communication 


Notes;  Item  numbers  refer  to  the  complete  item  descriptions  found  in  Appendix  B. 
Items  49  and  50  were  added  during  Round  2. 


Rating  Scale  for  Importance 


1 

2 

3 

4 

5 

Never 

Sometimes 

Important  about  as 

Often 

Always 

important 

important 

often  as  not 

important 

important 

Rating  Scale  for  Frequency 

1 

2 

3 

4 

5 

Never 

Sometimes 

Occurs  about  as 

Often 

Always 

occurs 

occurs 

frequently  as  not 

occurs 

occurs 

Unit  Designations 

1 

2 

3 

4 

5 

Squad 

Platoon 

Company 

Battalion 

Brigade 

Sample  Items  to  be  Rated: 

1 , Coordination  with  adjacent  units/commander. 

Imp,  scale:  1 2 3 4 5 Unit: Freq.  scale;  1 2 3 4 5 

2.  Integrated  utilization  of  supporting  arms  at  all  levels. 

Imp.  scale:  1 2 3 4 5 Unit: Freq.  scale:  1 2 3 4 5 

Figure  1 . Rating  scale  for  use  with  items  submitted  in  response  to  Question  1 . Round  One. 

a Instructions: 

“Question  2 was  open  to  misinterpretation  or  at  least  alternative  interpretations  - as  several  of 
you  pointed  out.  Despite  this,  the  responses  and  particularly  the  comments  provided  informa- 
tion on  the  basic  issue  - can  a unit  be  reliably  evaluated  in  less  than  the  total  time  available  in 
a field  training  exercise?  The  responses  indicate  two  basic  positions  — namely,  that  less  than 
the  full  time  is  needed  or  that  all  the  time  is  needed.  This  may,  however,  not  be  an  accurate 
assessment  of  your  position,  given  the  inadequacy  of  the  original  question.  We  arc,  therefore, 
providing  a complete,  verbatim  listing  of  the  comments  provided  on  the  question  and  asking 
for  your  response  on  the  same  question  based  on  a 96-hour  field  exercise  involving  a battalion(s) 
engaged  in  continuous  operations  with  both  offensive  and  defensive  phases.  Given  this  setting, 
how  much  time  out  of  96  hours  is  normally  needed  to  evaluate  (not  train)  each  of  the  units 
listed?  The  time  may  be  assumed  to  be  split  over  the  offensive  and  defensive  phases.” 

b.  Revised  question: 

"HOW  MUCH  TIME  - OUT  Of  96  HOURS  - IS  NORMALLY  NEEDED  TO  EVALUATE 
EACH  OF  THE  UNITS  LISTED?  THE  TIME  MA  Y BE  ASSUMED  TO  BE  DIVIDED 
BETWEEN  THE  OFFENSIVE  AND  DEFENSIVE  PHASES.  ” 
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3.  Question  3: 

Participants  were  given  a list  of  34  items  submitted  in  response  to  Question  3 during  Round  I (see 
Table  2 and  ApiJendix  C),  and  asked  to  rate  these  items  on  a rating  scale  (see  Figure  2).  Instructions  pro- 
vided and  the  original  question  are  shown  below. 

a.  Instructions: 

“The  following  items  were  submitted  in  response  to  Question  3 during  Round  One  of  the 
DELPHI.  You  are  asked  to  evaluate  the  items  listed  as  to  their  importance  in  accurately  eval- 
uating a unit,  the  unit  level(s)  at  which  they  are  most  applicable,  and  how  often  or  how  fre- 
quently the  item  is  incomplete  at  the  end  of  a field  exercise.  Rating  scales  are  provided  along 
with  a restatement  of  the  original  question.  Circle  or  check  the  number  on  the  importance  and 
frequency  rating  scales  and  fill  in  the  appropriate  number  or  numbers  in  the  unit  size  blank. 
Note:  Given  the  differences  between  combat  and  field  exercises  environments,  the  importance 
scale  as  used  here  refers  to  the  importance  of  the  item  or  factor  as  an  evaluative  item  and  as  an 
indicator  of  combat  readiness.  An  additional  page  is  provided  following  the  items  to  allow  you 
to  rephrase  items  and  to  add  additional  items.” 

b.  Question: 

"WHEN  YOU  ARE  ASSIGNED  TO  EVALUATE  A UNIT  IN  A FIELD  EXERCISE  ENVIRON- 
MENT. WHA  T INFORMA  TION  OR  FACTORS  ARE  USUALL  Y STILL  INCOMPLETE  A T 
THE  TIME  YOU  MAKE  OR  MUST  MAKE  YOUR  DECISION  CONCERNING  THE  UNIT’S 
COMB  A T READINESS?" 

TABLE  2.  SUMMARY  OF  DELPHI  ITEMS  DEVELOPED  FROM  QUESTION  3. 


1 . Ammunition 

2.  Armor  utilization 

3.  Close  air  support 

4.  Knowledge  of  supporting  arms 

5.  Effectiveness  of  supporting  arms 

6.  Operability  of  combat-essential  equipment 

7.  Reliability  of  combat-essential  equipment 

8.  Team  combat  functions 

9.  Individual  combat  skills 

1 0.  Commander’s  care  of  men 

1 1 . Commander’s  flexibility 

1 2.  Commander’s  professional  knowledge 

13.  Ability  to  communicate 

14.  Defensive  capability 

15.  Pcrsonnel/equipment  endurance 

1 6.  Fitne.ss 

17.  Unit  performance 


1 8.  Combat  execution 

19.  Live  fire  delivery 

20.  Fire/maneuverability 

21.  Organic  weapons 

22.  Fire  discipline 

23.  Status  information 

24.  Sustainment  ability 

25.  Timely  supply/resupply 

26.  Supply  economy 

27.  Intelligence 

28.  Equipment  casualties 

29.  Mission  accomplishment 

30.  Patrolling  capability 

31.  Personnel  stability 

32.  “Frago”  effectiveness 

33.  Expeditious  reporting 

34.  Reporting  procedures  (faulty) 

35.  Reporting  procedures  (timejy) 


Notes:  Item  numbers  refer  to  the  complete  item  descriptions  found  in  Appendix  C. 
Item  35  added  during  Round  Three. 
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Rating  Scale  for  Importance 


1 

Never 

important 

2 

Sometimes 

important 

3 

Important  about  as 
often  as  not 

4 

Often 

important 

5 

Always 

important 

1 

2 

Unit  Designations 
3 

4 

5 

Squad 

Platoon 

Company 

Battalion 

Brigade 

Evaluative  Completeness 

1 

2 

3 

4 

5 

Never 

Sometimes 

Incomplete  about  as 

Often 

Always 

:omplete 

incomplete 

frequently  as  not 

incomplete 

incomple 

Figure  2.  Rating  scale  for  use  with  items  submitted  in  response  to  Question  3,  Round  One. 

4.  Question  4: 

A fourth  question  was  added  in  Round  Two.  It  was  presented  in  a. format  similar  to  that  used  in  the 
first  round: 

"WHAT  FACTORS  INFLUENCE  THE  ABILITY  OF  A UNIT  TO  ACCOMFIISII  ITS  HUSSION 
BUT  ARE  NORMALLY  BEYOND  THE  IMMEDIATE  CONTROL  OF  THE  UNIT  COM- 
MANDER? E.G.,  TERRAIN.  SUPPORT.  ETC. ...” 


ROUND  THREE 

Round  three  materials  were  distributed  to  participants  as  in  the  second  round.  General  information 
reflecting  Round  Three  requirements  was  provided  (see  Appendix  A). 

1 . Questions  1 and  3: 

Means  and  standard  deviations  on  the  items  derived  from  Questions  1 and  3 in  Round  Two  were 
provided  to  participants  as  feedback,  along  with  histograms  of  the  unit  applicability  distributions.  General 
comments  submitted  were  used  to  preface  the  questions,  and  specific  comments  were  placed  with  the  items 
to  which  they  referred. 


2.  Question  2: 

Completed  and  withdrawn  from  further  consideration. 

3.  Question  4; 

Participants  were  given  a list  of  1 7 items  submitted  in  response  to  Question  4 during  Round 
Two  (sec  Table  3 and  Appendix  D),  and  asked  to  rate  them  according  to  importance  and  unit  applicability. 
The  question  was  formatted  similar  to  Questions  1 and  3 in  the  second  round. 

RC'JND  FOUR 

Materials  were  sent  to  participants  as  in  the  previous  round.  Question  1 was  considered  completed 
and  withdrawn  from  further  consideration  along  with  all  but  seven  selected  items  from  Question  3 (Nos.  1 , 
11,  17.  18,  32, 33  and  34)  on  the  basis  of  a diminishing  rate  of  change  in  the  standard  deviation  from  the 
previous  round.  These  seven  items  along  with  the  Question  4 items  were  formatted  as  in  the  previous  round 
General  information  provided  on  Round  Four  is  found  in  Appendix  A. 

TABLE  3.  SUMMARY  OF  DELPHI  ITEMS  DEVELOPED  FROM  QUESTION  4 


1 . Terrain/hydrography 

2.  Weather 

3.  Time 

4.  Space 

5.  Intelligence 

6.  Support 

7.  Fatigue  level 

8.  Rules  of  engagement 

9.  Resources 


10.  Overcontrol 

1 1 . Civilians  in  zone 

1 2.  Peacetime  safety  requirements 

13.  Replacements 

14.  Equipment  reliability 

15.  Medical  problems 

1 6.  Long-range  communication 

17.  Support  base  distance 


Note;  Item  numbers  refer  to  the  complete  item  descriptions  contained  in  Appendix  D. 


RESULTS 

Tlie  results  are  organized  by  question  and  round.  The  number  of  participants  arc  as  follows:  Round 
One.  1 1 (44%);  Round  Two,  14  (56%);  Round  Three,  10  (40%);  and  Round  Four,  1 1 (44%). 


QUESTION  1 

"WHEN  YOU  ARE  ASSIGNED  TO  EVALUATE  A UNIT  IN  A FIELD  EXERCISE  ENVIRON- 
MENT, WHAT  FACTORS  OR  CHARACTERISTICS  DO  YOU  USE  IN  FORMING  YOUR 
EVALUATIONOF  THE  UNIT’S  COMBAT  READINESS?’’ 


ROUND  ONE 

A total  of  121  items  was  submitted  in  response  to  Question  1 . This  was  reduced  to  48  by  combining 
redundant  items  and  deleting  those  beyond  the  scope  of  TWAES  project  requirements  (two  additional  items 
added  in  Round  Two  brought  the  total  to  50).  These  items  are  listed  in  Appendix  B.  Item  breakdown  by 
unit  level  applicability  is  presented  in  Appendix  E. 


ROUND  TWO 

Data  were  obtained  on  the  importance  and  frequency  of  each  item,  in  addition  to  the  unit  levels  at 
which  each  was  most  applicable.  Means  and  standard  deviations  for  the  importance  scale  and  the  unit  appli- 
cability graphs  arc  presentedan  Appendix  B. 


ROUND  THREE 

Round  Three  consisted  of  42  items.  Low  importance  levels,  high  standard  deviations,  and  participant 
comments  influenced  the  deletion  of  items  7, 15,  21, 28, 30, 40, 41 , and  48  from  Round  Three.  Items  49 
and  50  were  submitted  by  a participant  during  Round  Two.  Appendix  B shows  the  final  means,  standard 
deviations,  and  unit  applicability  graphs  for  each  item.  A number  of  the  participants  chose  not  to  respond  to 
Question  1 in  this  round.  Therefore,  the  mean  response  to  an  item  from  Round  Two  was  used  for  missing 
responses  in  computing  Round  Three  means.  This  is  in  accordance  with  the  initial  instructions  to  the 
DELPHI  participants  providing  the  option  of  not  respondingrto  a particular  question  if  they  did  not  wish  to 
revise  their  previous  estimate. 


ROUND  FOUR 

Completed  and  withdrawn  from  further  consideration. 


QUESTION  2 

ROUND  ONE 

Question  2 as  presented  in  the  first  round  caused  a wide  range  of  responses  along  with  extensive  com- 
ments, indicating  ambiguity  in  the  question  and  response  format  and  inadequate  problem  parameters.  The 
question  was  reformatted  for  the  following  round. 
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ROUND  TWO 

The  question  and  parameters  were  reformulated  for  this  round  as  follows: 

"HOW  MUCH  TIME  - OUT  OF  96  HOURS  - IS  NORMALLY  NEEDED  TO  EVALUATE 
EACHOFTHE  UNITS  LISTED?  THE  TIME  MAY  BE  ASSUMED  TO  BE  DIVIDED 
BETWEEN  THE  OFFENSIVE  AND  DEFENSIVE  PHASES. " 

Tlie  parameters  of  the  questions  were  established  by  tlie  following  statement: 

. . based  on  a 96-hour  field  exercise  involving  a battalion(s)  engaged  in  continuous  operations 
. . . how  much  time  ...  is  needed  to  evaluate  (not  train)  each  of  the  units  listed?” 

The  responses  to  this  question  for  both  Round  One  and  Round  Two  are  presented  in  Table  2,  which  includes 
the  ranges,  means,  and  the  distributions  of  the  responses  in  hours.  The  responses  are  not  strictly  comparable 
as  the  question  was  modified,  but  the  important  aspect  is  the  improved  consensus.  The  high  degree  of  con- 
sensus obtained  allowed  this  question  to  be  withdrawn  from  succeeding  rounds.  The  results  supported  the 
assumption  that  smaller  units  require  the  services  of  evaluative  personnel  for  less  time  than  larger  units. 

ROUND  THREE 

Completed  and  withdrawn  from  further  consideration. 
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TABLE  4.  QUESTION  2 RESPONSE  SUMMARY 
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ROUND  ONE 


ROUND  TWO 


Range  Range  Mean  Distribution 
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Brigade  96-168  72-96  96 


Battalion  4-96  48-96 


Company  1-48  24-67 


Platoon  0.5-48  12-48 


Squad  0.25-48  8-48 


Note:  Responses  are  given  in  hours. 
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QUESTION  3 

"WHEN  YOU  ARE  ASSIGNED  TO  EVALUATE  A UNIT  IN  A FIELD  EXERCISE  ENVIRON- 
MENT, WIIA  T INFORMA  TION  OR  FACTORS  ARE  USUALL  Y STILL  INCOMPLETE  A T 
THE  TIME  YOU  MAKE  OR  MUST  MAKE  YOUR  DECISION  CONCERNING  THE  UNIT’S 
COMB  A T READINESS?  ” 


ROUND  ONE 

Forty-three  items  were  submitted  in  response  to  Question  3.  The  deletion  of  duplicate  items  and 
items  beyond  the  scope  of  TWAES  reduced  this  number  to  34  (an  additional  item  added  in  Round  Three 
brought  the  total  to  35).  These  items  are  listed  in  Appendix  C. 


ROUNDTWO 

Data  were  collected  on  each  item’s  importance,  frequency  of  incompleteness,  and  the  level  of  unit 
applicability.  Means  and  standard  deviations  for  the  importance  and  evaluative  completeness  scales  are  pre- 
sented in  Appendix  C,  along  with  unit  applicability  graphs. 


ROUNDTHREE 

The  means  and  standard  deviations  of  the  reevaluated  items  are  also  shown  in  Appendix  C.  Overall, 
the  SD  of  the  importance  scale  was  reduced,  whileThe  SD  of  the  evaluative  completeness  scale  generally 
increased.  The  histograms  depicting  levels  of  unit  applicability  showed  little  changeTrom  the  previous 
round.  An  additional  item  - No.  35  - was  submitted  in  this  round. 


ROUNDFOUR 

Items  1, 11, 17, 18,  32,  33,  and  34,  along  with  the  new  item  (No.  35)  submitted  in  the  previous 
round,  were  selected  for  further  evaluation  on  the  basis  of  a considerable  change  in  responses  from  the  pre- 
vious round.  In  reference  to  the  importance  scale,  Appendix  C shows  a decrease  in  item  variability,  except 
for  items  1 and  34.  In  the  completeness  scale  for  items  17  and  18,  both  mean  responses  and  variability  were 
considerably  smaller. 


QUESTION  4 

"WHAT  FACTORS  INFLUENCE  THE  ABILITY  OF  A UNIT  TO  ACCOMPLISH  ITS  MISSION 
BUT  ARE  NORMALLY  BEYOND  THE  IMMEDIATE  CONTROL  OF  THE  UNIT  COM- 
MANDER? E.G.,  TERRAIN.  SUPPORT.  ETC. ...” 


ROUNDTWO 

A total  of  97  items  was  submitted  in  response  to  Question  4.  Many  of  these  items  were  related  to 
personnel  rather  than  to  the  requirements  of  the  TWAES  project.  Appendix  D lists  the  17  items  which  were 
retained  for  evaluation  after  the  duplicates  and  personnel-related  items  were  deleted. 


ROUND  THREK 

In  contrast  to  Questions  1 and  3,  higher  levels  of  variability  were  observed  on  the  importance  ratings 
of  the  items  in  Question  4 (see  Appendix  D).  Mean  ratings  as  low  as  3.30  were  found,  as  is  shown  on  items 
15  and  Id. 


ROUND  FOUR 

A general  decrease  in  standard  deviations  was  observed,  while  mean  responses  tended  to  increase  {see 
Appendix  D).  (Mean  and  SD  data  were  not  collected  on  items  13  and  14.)  These  statistics  seem  to  indicate 
a liigli  degree  of  consensus  among  the  participants  on  these  items.  Further  information  on  unit  level  was  not 
solicited  for  this  round.  Therefore,  the  unit  results  shown  are  Round  Three  results.  Item  breakdown  by  unit 
level  applicability  is  presented  in  Appendix  E. 


DISCUSSION 

Items  related  to  unit  performance  evaluation  were  developed  through  the  use  of  the  DELPHI.  These 
Items  should,  ideally,  represent  the  key  factors  in  the  evaluative  policy  of  the  officer  participants.  “Ideally" 
is  used  parenthetically  because  it  is  impossible  to  know  if  this  is  true  in  an  absolute  sense.  The  DELPHI 
technique  itself  has  well-established  validity  and  the  participants  were  welkjualified  experts. 

TJie  participants  were  Marine  Corps  infantry  battalion  commanders  in  the  ranks  of  Major  tlirough 
Colonel.  Their  tenure  in  the  Marine  Corps  ranged  from  15  years  to  28  years,  with  a mean  of  22  years.  Since 
not  all  of  the  officers  who  participated  responded  in  all  rounds,  a comparison  was  made  of  the  responses  to 
Questions  i and  3 by  those  who  participated  in  both  rounds  one  and  two  and  those  who  participated  initially 
in  round  two.  A t-test disclosed  no  significant  differences. 

Some  of  the  procedural  details  of  this  DELPHI  varied  from  conventional  procedures  of  previously 
reported'DELPHl.  These  variations  are  described  with  respect  to  possible  effects  on  the  validity  of  the  ob- 
tained results.  In  the  typical  DELPHI,  participation  by  experts  is  constant  from  round  to  round.  However, 
in  this  case,  due  to  individual  circumstances  and  the  nature  of  the  field,  this  was  not  achieved  ~ the  group 
membership  was  fluid.  For  example,  five  commanders  initiated  their  participation  in  the  second  round.  Of 
these  five,  only  two  submitted  round  tliree  data.  A further  deviation  from  conventional  DELPHI  was  that 
the  experts  were  not  consulted  prior  to  the  exercise  and,  consequently,  did  not  formally  agree  to  participate. 
Instead,  their  participation  was  taken  as  evidence  of  agreement.  This  approach  was  seen  as  a tradeoff  be- 
tween obtaining  compliance  and  participation  and  was  discussed  with  Marine  Corps  project  representatives 
prior  to  the  exercise.  The  level  and  quality  of  participation  achieved  were  probably  due  (T)  to  the  inherent 
interest  of  the  participants  in  the  performance  evaluation  area,  and  (2)  the  fact  that  the  TWAES  is  rapidly 
being  implemented  and  this  exercise  provided  one  means  for  users  to  influence  its  final  configuration.  The 
foregoing  is  intended  to  minimize  neither  the  exercise  nor  the  results,  but  to  show  how  the  DELPHI  was 
necessarily  adapted  for  this  particular  application  and  some  of  the  assumptions/compromises  which  were 
made. 

A basic  assumption  of  the  DELPHI  procedure  is  that  participant  responses  wilimerge  with  each  suc- 
cessive round,  e.g.,  members  will  take  into  consideration  the  average  group  response  when  they  submit  their 
new  responses.  Support  for  this  assumption  was  shown  by  the  standard  deviation  measures  on  the  items  in 
Questions  1 , 3,  and  4.  For  the  majority  of  these  items,  the  standard  deviations  become  successively  smaller 
with  each  round.  Question  2 variability  was  sharply  reduced  following  introduction  of  the  feedback  and  the 
more  specific  problem  parameters. 
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TIk’  evaluative  items  submitted  provide  a range  of  application  from  squad  level  to  brigade  level. 
Somewhat  more  items  were  provided  at  the  company  and  battalion  levels.  This  may  reflect  the  unit  levels 
of  primary  importance  in  evaluation  or,  in  part,  the  bias  of  the  senior  level  officers  who  participated.  This 
question  will  require  further  definition.  Questions  5 and  3 provided  information  on  evaluative  items  which, 
while  important,  may  not  always  be  observable  for  complete  evaluation.  Ideally,  adjustments  in  scenarios 
might  be  made  to  allow  more  complete  evaluation  of  such  areas.  If  this  proves  impractical,  the  use  of  such 
items  in  an  evaluative  schema  would  be  kept  to  a minimum. 

The  items  submitted  in  response  to  Questions  1 and  4 were  amenable  to  categorization  by  several 
strategies.  Tentative  categorization  was  achieved  by  determining  whether  the  item  was  primarily  command, 
team,  or  individual  performance.  Within  the  command  category,  further  categorization  was  possible  in 
terms  of  tactical  decision  items  and  administrative  items.  This  was  particularly  true  at  the  company,  bat- 
talion, and  brigade  levels  of  application.  This  is  consistent  with  research  by  Hclme,  Willemin,  & Grafton 
( 1971 ).  who  found  that  "two  broad  domains  of  officer  activity  were  clearly  differentiated  - combat  leader- 
ship and  technical/managerial  leadership.”  This  breakdown  of  items  also  lends  itself  to  an  evaluation  method 
developed  by  Cook  and  Baker  (1968)  for  use  with  armored  cavalry  units.  That  method  concentrated  on 
evaluating  the  unit  leader’s  skill  in  command  and  the  unit’s  performance  in  executing  those  commands.  A 
tentative  breakdown  of  items  for  Questions  1 and  4 by  these  means  is  shown  in  Appendix  E. 

The  environmental  items  submitted  for  Question  4 can  be  roughly  classified  into  those  that  ( 1 ) refer 
to  the  physical  environment  and  (2)  are  of  a situational  nature.  Situational,  as  used  here,  refers  to  factors 
influencing  the  outcome  of  an  engagement  that  resulted  from  previous  decisions  or  events  but  which  are  not 
under  the  immediate  control  of  the  unit  commander.  Question  4 generated  more  comment  per  number  of 
items  submitted  than  did  Questions  1 and  3,  Three  major  areas  of  interest  or  controversy  developed  which 
concerned  ( 1 ) what-is  under  control,  (2)  the  issue  of  overcontrol,  and  (3)  personnel  problems.  The  issue  of 
how  much  is  under  the  control  of  the  unit  commander  arose  in  one  specific  case  over  fatigue  level  and  also 
as  a general  response  to  the  entire  question.  A minority  sentiment  expressed  was  that  all  factors  are  under 
control  if  command  is  adequate  and  that  Question  4 was  unnecessary.  The  issue  of  overcontrol  by  higher 
headquarters  had  two  distinct  facets.  The  first  and  mostobvious  was  command-type  overcontrol  by  superior 
officers  of  tasks  that  should  have  been  handled  at  a lower  level.  This  may  have  the  effect  of  reducing  flexi- 
bility at  lower  levels  by  subordinates.  To  the  extent  that  this  issue  is  a problem,  it  should  be  reflected  in 
unit  performance  and  be  measurable  by  TWAES. 

The  second; facet  of  overcontrol  is  of  more  interest  to  performance  evaluation  and  TWAES  - that  is, 
overcontrol  and  oversupervision  of  exercises  as  a whole  which  can  alter  the  conduct  of  an  exercise.  Close 
supervision  can  remove  unwanted  variability  in  an  exercise  situation  and  can  make  exercises  easier  to  con- 
duct. Oversupervision,  however,  in  the  interest  of  reducing  variability,  results  in  reducing  information  and 
response  loads  at  all  levels,  which  in  turn  reduces  the  authenticity  of  the  situation  and  the  accuracy  of  the 
performance  evaluation  (Arima,  1969). 

The  third  area  of  controversy  concerned  personnel-related  factors  over  which  the  commander  has  no 
control  but  which  may  reduce  unit  performance.  Examples  of  the  items  submitted  were  high  levcls  of  per- 
sonnel turnover  prior  to  embarkation,  drug  problems,  racial  problems,  short  duration  of  exercise,  etc.  The 
very  large  number  of  responses  of  this  type  indicated  the  concern  of  the  participants.  These  items  were  not 
pursued,  however,  because  any-performance  degradation  they  cause  should  be  measureable  by  the  TWAES 
performance  evaluation  capability.  To  the  extent  that  these  factors  continue  to  adversely  affect  perform  .nee, 
efforts  should  be  made  to  ameliorate  them.  However,  this  is  not  within  the  scope  of  the  TWAES  project. 

The  responses  to  Question  2 were  in  accord  with; the  expectation  that  small  units  should  require  less 
time  to  evaluate  than  large  ones.  The  relatively  short  period  of  time  needed  for  assessments  at  lower  levels 
could  permit  the  use  of  trained  evaluators  who  could  move  from  unit  to  unit  during  an  exercise,  leaving  the 
routine  control  functions  to  less  specialized  individuals. 
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Qucj-t.ons  1 , 3,  and  4 have  statistics  associated  witn  each  item.  Except  for  the  unit  applicability 
j’rai)hs.  the  statistics  served  primarily  for  control  purposes  during  the  conduct  of  the  DELPKI  and  only 
secondarily  for  estimating  the  relative  value  of  any  item.  For  purposes  of  developing  a performance  evalua- 
tion system,  all  of  the  items  are  initially  considered  initx)rtant  and  as  basic  elements  to  be  developed  and 
integrated.  The  items  will  be  refined  in  later  stages  of  the  research  into  behaviorally-oriented  items  for 
improved  observability  by  exircise  evaluators  of  lesser  experience  than  that  of  the  participants  who  devel- 
oped the  items. 

The  items  developed  in  Questions  1 , 3,  and  4 will  be  used  in  the  next  phase  of  the  research  to  develop 
specific  performance  evaluation  items.  These  items  will  be  grouped  first  by  the  level  of  unit  applicability 
and  then  into  command  decision,  team  performance,  and  individual  performance  areas,  as  is  shown 
in  Appendix  U.  Further  breakdowns  within  these  areas  are  possible,  depending  on  the  task  requirements. 
These  groups  of  items  will  be  examined,  statistically,  from  unit  level  to  level  for  the  effects  of  subsystem 
performance  on  system  performance.  This  approach  will  allow  some  measure  of  the  embedded  performance 
event  to  be  made.  Embedded  events,  as  used  here,  refer  to  events  in  which  a subsystem  (squad  or  platoon) 
mission  is  an  important  factor  in  the  success  of  a larger  overall  mission.  This  approach  will  provide,  in  an 
ideal  sense,  some  measure  of  the  contribution  of  each  unit’s  performance  to  the  whole.  Similarly,  examining 
the  effect  of  environmental  variables  on  unit  performance  will  provide  a means  of  factoring  out  the  effect 
of  environment  and  thereby,  ideally,  allow  the  normalizations  of  unit  performance  scores.  Data  for  these 
items  must  be  derived,  at  least  in  part  empirically,  to  provide  validity  of  the  relationships. 


CONCLUSIONS  AND  RECOMMENDATION 

The  time  requirements  for  exercise  evaluations  were  established  in  terms  of  the  average  times  needed 
to  evaluate  units  varying  in  size  from  squad4o  brigade.  The  specified  time  requirements  indicate  that  lower- 
level  units  can  be  evaluated  in  a relatively  short  time  by  the  highly-trained  evaluators  on  a rotating  basis, 
leaving  the  control  function  to  lesser  trained/experienced  personnel. 

Performance  response  items  for  use  in  evaluating  tactical  field  exercises  were  developed.  Addition- 
ally, data  were  collected  on  performance  items  which,  while  important,  may  occur  too  infrequently  to  be 
used  in  an  evaluative  system.  Contextual  factors  (physical  and  situational)  which  influence  performance 
' were  similarly  identified. 

An  initial  evaluation  model  incorporating  both  the  contextual  and  the  response  factors  has  been 
developed.  This  tentative  model  will  be  further  developed  through  the  collection  and  statistical  analyses 
of  field  performance  data  to  establish  empirically  the  direction  and  degree  of  relationship  between  the  sets 
of  context  and  performance  variables.  When  sufficient  empirical  data  have  been  collected,  normalized  unit 
performance  indices  will  be  derived  to  provide  unit  commanders  with  expected  values  of  unit  performance 
for  giveivsets  of  contextual  factors.  This  evaluation  system,  by  separating  the  effects  of  environment  from 
unit  performance  responses,  will  eventually  permit  the  direct  comparison  of  performance  across  similar  types 
of  units  irrespective  of  mission  environments. 

It  is  recommended  that  the  performance  contextual  and  response  items  be  integrated  into  the  exer- 
cise evaluation  format,  and  thatinitial  reliability  and  validity  data  be  gathered  from  the  field. 
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PROJECT  AND  GENERAL  INFORMATION 


PROJECT  INFORMATION 

The  Navy  Personnel  Rcsearclv  and  Development  Center  has  been  tasked  by  the  Marine  Corps  Devel- 
opment and  Education  Command  to  develop  improved  unit  performance  measures  and  criteria  for  use  in 
conjunction  with  the  Marine  Corps  Tactical  Warfare  Analysis  and  Evaluation  System  (TWAES).  As  part  of 
this  project,  an  effort  is  being  made  to  utilize  the  considerable  experience  of  senior  field  grade  officers  for 
the  development  of  these  needed  performance  measures  and  criteria.  There  is  a requirement  to  identify 
those  characteristics  of  unit  performance  that  distinguish  a combat-ready  unit  from  a noncombat-ready  unit 
as  observed  in  a field  exercise  environment.  Some  of  the  distinguishing  characteristics  or  actions  arc  largely 
subjective  m nature  and,  as  such,  are  difficult  to  define,  observe,  and  quantify.  To  meet  this  problem,  the 
DELPHI  tech,  que  developed  by  the  Rand  Corporation,  which  provides  a systematic  means  for  extracting 
expert  opinion,  will  be  used  in  collecting  and  compiling  the  needed  information  on  unit  performance 

You,  along  with  all  other  infantry  battalion  commanders,  as  experts  in  the  practical  field  evaluation 
of  combat  readiness,  are  asked  to  provide  and  judge  unit  performance  measures  and  criteria  for  ultimate  use 
with  the  TWAES  system.  Your  contributions  of  time  and  effort  to  this  phase  of  the  TWAES  project  will 
contribute  to  the  success  of  this  task. 


DELPHI:  GENERAL  INFORMATION 

In  the  DELPHI  procedure,  a scries  of  questionnaires  is  given  to  a group  of  respondents  to  determine 
the  group’s  views  on  a particular  topic.  As  a participant,  you  should  discard  all  traditional  theories  of  sur- 
veys and  questionnaires  when  approaching  this  exercise.  This  is  not  an  attempt  to  design  a short-answer, 
"quickie''  questionnaire.  Rather,  we  encourage  you  to  think  deeply  about  the  questions,  and  to  formulate 
new  questions  and  explore  new  alternatives.  We  ask  you,  as  a respondent,  to  spend  a total  of  4 or  5 one-half 
days  distributed  over  approximately  4 months. 

The  DELPHLapproach  is  cumulative.  The  first  round  is  largely  exploratory  and  designed  to  open  up 
new  areas  of  thought.  As  the  second  and  third  rounds  progress,  the  areas  of  interest  are  narrowed  and  group 
views  on  certain  topics  begin  to  emerge.  After  each  round,  the  reasoning  of  participants  is  fed  back  to  all 
respondents  for  their  appraioal.  By  the  fourth  or  fifth  round,  there  is  a clear  indication  of  the  group's  opin- 
ions and  attitudes  on  the  subject  of  the  exercise.  It  should  also  be  emphasized  that,  unlike  the  traditional 
one-shot  survey  technique,  we  are  dealing  largely  with  factors  that  can  not  be  quantified  by  analysis. 

All  necessary  steps  will  be  taken  to  preserve  the  anonymity  of  the  respondents,  this  being  an  essential 
feature  of  this  technique. 

Questionnaires  in  a DELPHI  exercise  are  meant  to  be  only  a stimulus,  rather  than  a straightjacket. 

As  a respondent,  you  have  the  following  options  with  respect  to  any  particular  question,  subquestion  or 
alternatives  presented: 

1 . You  may-choose  not  to  answer  a question  if  you  feel  your  judgment  would  be  “risky  .”  In  this 
risky  situation,  if  you  feel  the  judgment  is  an  important  one  to  be  made,  we  would  hope  for  a comment  on 
the  type  of  information  that  would  aid  you  or  others  in  making  the  particular  judgment. 

2.  You  may-choose  to  rewrite,  as  a comment,  a particular  question  and  then  answer  your  version  if 
you  feel  the  original  is  misleading. 

3.  You  may  suggest  questions  you  would  like  to  see  in  the  next  questionnaire  if  you  feel  they  would 
clarify  an  issue  or  raise  a new  alternative  that  the  group  should  consider. 

4.  You  may  consult  with  associates,  but  not  with  other  participants,  on  specific  items  in  the  ques- 
tionnaire or  utilize  other  information  sources  normally  available  to  you.  However,  you  should  respond  as  an 
individual  and  not  as  a spokesman  fora  particular  group. 
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5.  You  may,  and  are  encouraged,  to  express  short  arguments  or  comments  on  any  judgment  about 
whicli  you  feel  confident.  This  may  include  items  of  information  that  lead  you  to  the  judgment.  The  short- 
er the  comment,  the  greater  the  probability  that  it  will  be  included  intact  in  the  summary  which  is  fed  back 
to  the  participants  on  the  next  round. 

Whenever  a consensus  is  obtained  on  an  issue,  we  will  usually  drop  that  issue  from  further  exploration  in 
succeeding  questionnaires.  When  a polarization  of  views  occurs,  wc  will  attempt  to  develop  questions  de- 
signed to  highlight  reasons  for  the  polarization.  The  degree  to  which  the  design  team  chooses  to  explore  a 
majority-minority  type  [wlarization  will  be  more  a function  of  the  arguments  or  comments  made  tnan  of  the 
actual  number  of  individuals  taking  a particular  view. 


INSTRUCTIONS  FOR  PARTICIPATION  IN  THE  DELPHI:  ROUND  ONE 

The  study  in  which  you  are  about  to  participate  is  for  the  purpose  of  investigating  behavior  that  is 
characteristic  of  a combat-ready  unit  and  of  the  evaluative  decision  processes  of  the  commanding  officer. 
Your  primary  task  is  to  answer  each  question  based  on  your  experience  and  knowledge.  You  may  choose 
not  to  respond  to  a particular  question.  Further  instructions  are  as  follows; 

1 . Please  return  the  completed  questionnaire  within  5 days  after  receiving  it. 

2.  Please  do  not  discuss  your  participation  in  this  study  or  the  contents  thereof  with  other 
participants. 

3.  Do  not  hesitate  to  contact  members  on  the  design  team  by  telephone  if  you  need  clarification  on 
the  questions  or  instructions:  (714)  225-2371,  Autovon  933-2371. 

You  can  expect  this  DELPHI  to  consist  of  four  or  five  rounds.  Therefore,  you  a.c  requested  to  pro- 
vide your  name  and  address  and  to  apprise  the  design  team  of  any  change  of  address  sc  v/e  may  keep  in 
contact  with  you  for  the  duration  of  the  study.  We  again  emphasize  the  fact  that  your  name  will  not  be 
identified  with  the  materials  and  comments  you  submit  nor  will  your  name  be  known  or  released  to  anyone 
beyond  the  design  team.  Please  return  this  page  and  the  response  pages  in  the  enclosed  envelopes. 


DELPHI:  ROUND  TWO 

The  purpose  of  the  second  round  of  the  DELPHI  is  to  inform  you  of  the  items  submitted  by  the 
other  participants  and  to  allow  you  the  opportunity  to  revise,  eliminate,  or  add  items.  Additionally,  a 
fourth  question  is  added  for  your  consideration. 

The  items  submitted  for  Question  I are  listed  essentially  as  they  were  received  except  in  those  cases 
where  more  than  one  participant  submitted  similar  items.  In  those  cases,  the  items  were  combined  when  no 
change  of  meaning  or  emphasis  seemeddikely  to  result.  If  you  see  an  item  that  no  longer  reflects  your  posi- 
tion, you  arc  asked  to  correct  it. 

All  of  the  responses  submitted  for  Question  2 are  presented  to  show  the  comments  of  the  participants 
and  the  range  and  frequency  of  the  responses  submitted.  As  noted  in  the  instructions  for  Question  2,  the 
question  has  been  altered  to  incorporate  some  of  your  suggestions. 


DELPHI:  ROUND  THREE 

The  third  round  of  the  DELPHI  is  to  inform  you  of  the  responses  submitted  by  the  other  participants 
and  to  allow  you  the  opportunity  to  revise,  eliminate,  or  add  items.  While  Questions  1 , 3 and  4 have  been 
revi.sed  for  Round  Three,  the  concluding  results  to  Questions  2 are  presented. 
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The  responses  submitted  for  Question  1 are  summarized  and  presented  along  witli  any  comments 
received.  A few  items  were  dropped  because  of  low  importance  or  duplication  of  items.  Question  2 was 
completed  with  the  last  round  and  a summary  of  that  data  is  provided  along  with  space  for  comments  you 
may  have.  Question  3 presents  an  interesting  problem  in  that  many  of  the  items  received  widely  diverse 
responses  in  terms  of  evaluative  completeness.  All  of  the  responses  submitted  and  the  comments  received 
arc  summarized.  This  question  needs  special  attention  and  comments.  Question  4 is  made  up  of  the  items 
received  on  the  last  round. 


DELPHI;  ROUND  FOUR 

The  fourth  round  of  the  DELPHI  is  to  inform  you  of  the  responses  submitted  by  the  other  partici- 
pants in  Round  Tliree  and  to  allow  you  to  revise,  eliminate,  or  add  items.  Only  Questions  3 and  4 will  be 
considered  at  this  time.  A complete  summary  of  all  questions  will  be  sent  when  this  round  is  completed  and 
analyzed. 

A few  items  have  been  selected  from  Question  3 for  reevaluation  in  this  round.  Since  a consensus 
was  obtained  on  the  remaining  items,  they  will  not  be  reconsidered.  A summary  of  responses  and  comments 
to  Question  4 is  presented.  All  of  these  items  will  be  reevaluated  in  this  round. 

For  all  practical  purposes,  this  will  be  the  last  round  in  which  you  will  be  asked  to  respond  to 
DELPHI  questions.  It  is  asked  that  you  consider  each  item  carefully  and  provide  comments  in  support  of 
your  responses  where  appropriate. 


QUESTION  ! RESULTS 


ITEMS 


1 . Coordination  with  adjacent 
imits/conimander. 

Importance 

Mean 

Standard  Deviation 

2.  Integrated  utilization  of  sup- 
porting arms  at  all  levels. 

Importance 

Mean 

Standard  Deviation 

3.  Exercise  of  noise,  light,  disper- 
sion, police,  and  local  security 
discipline. 

Importance 

Mean 

Standard  Deviation 

4.  Timeliness  with  which  casual- 
ties are  evacuated. 

Importance 

Mwn 

Standard  Deviation 

5.  Ability  to  concentrate  combat 
power  at  the  decisive  time  and 
place  through  fire,  maneuver, 
and  awareness  of  time/space 
factors. 

Importance 

Mean 

Standard  Deviation 


QUESTION  1 RESULTS  (CONT.) 


ROUND 

ITEMS 

TWO 

THREE 

6. 

Ability  to  aiialyze/uiiderstand 
mission  and  terrain. 

Importance 

Mean 

4.60 

4.50 

Standard  Deviation 

0.70 

0.64 

*7. 

Ability  to  timely  and  effectively 
discipline  command  ix)st/staff. 

Importance 

Mean 

3.60 

Standard  Deviation 

1.20 

8. 

Ability  of  staff^to  coordinate, 
and  to  respond-to  others/ 
situation  in  a timely  manner. 

Importance 

Mean 

4:80 

4.90 

Standard  Deviation 

0.50 

0.11 

9. 

Ability  of  commander  to  issue 
clear,  concise,  timely,  and 
complete  orders. 

Importance 

Mean 

4.70 

4.60 

Standard  Deviation 

0.70 

0.63 

10. 

Ability  of  commander  and  staff 
to  pace  themselves. 

Importance 

Mean 

4.00 

3.88 

Standard  Deviation 

0.80 

0.83 

% UNIT  APPLICABILITY 


ROUND 

TWO 


to  Cun  SB 


ROUND 

THREE 


g|sSa 


’•'Deleted  from  Round  Three 
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QUESTION  I RESULTS  (CONT.) 


1 1 . Ability  to  communicate  by 
primary  and  alternative  means 
in  a timely,  brief,  accurate,  and 
secure  manner. 

Importance 


Mean 

Standard  Deviation 

12.  Effectiveness  of  encryption 
and  shackling  and  use  of 
authentication. 

Importance 


Mean 

Standard  Deviation 

13.  Ability  to  control  forces  and 
utilize  proper  troop  leading 
steps  at  all  levels. 

Importance 


Mean 

Standard  Deviation 

1 4.  Methods  and  effectiveness  of 
control  by  small  unit  leaders. 

Importance 

Mean 

StandardDcviation 

*15.  Knowledge  of  general  and 

special  situations  by  small  unit 
leaders. 

Importance 


Mean 

Standard?Deviations 


ROUND 

TWO 

THREE 

4.50 

4.30 

0.90 

0.61 

4.10 

3.80 

0.90 

0.69 

4.60 

4.30 

0.70 

0.60 

4.50 

4.50 

0.70 

0.38 

4.00 

1.20 

% UNIT  APPLICABILITY 


ROUND 

TWO 


ROUND 

THREE 


be 
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QUESTION  1 RESULTS  (CONT.) 


ROl 

UND 

ITEMS 

TWO 

THREE 

1 6.  Ability  of  unit/commandcr  to 
utilize  logistical  and  service 
support. 

Importance 

Mean 

4.30 

3.80 

Standard  Deviation 

1.00 

0.79 

17.  Ability  of  a unit  to  sustain 
itself  operationally  and 
logistically. 

Importance 

Mean 

4.60 

3.90 

Standard  Deviation 

0:70 

0.89 

18.  Knowledge/display  of  defensive 
principles  (digging-in,  sectors 
of  fire,  etc. . . .). 

Importance 

Mean 

4.70 

4.90 

Standard  Deviation 

0.70 

0.16 

19.  Completeness  and  condition  of 
organizational  equipment. 

Importance 

Mean 

4.40 

4.40 

StandardiDcviation 

0;80 

0.39 

20.  Completeness  and  condition  of 
individual  equipment. 

Importance 

Mean 

4.30 

4.20 

StandardDeviation 

1.00 

0.59 

% UNIT  APPLICABILITY 


ROUND 

TWO 


be 

c rt.2 

■o  SS.'s'S 

3-eS  S, 
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I I 
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Squad 

Platoon 

Company 

Battalion 

Brigade 


QUESTION  1 RESULTS  (CONT.) 


ROUND 


TWO  THREE 


% UNIT  APPLICABILITY 


ROUND  ROUND 

TWO  THREE 


c 

c 5 .2 

■a  gS.^'S 

s|eS.§, 

(?Eua£ 


*2 1 . Exercise  of  fire  discipline. 
Importance 


Mean 

Standard  Deviation 

22.  Time  from  request  for  fire  sup- 
port to  delivery. 

Importance 

Mean 

Standard  Deviation 

23.  Proper  flow  of  information  be- 
tween higher,  adjacent,  and 
subordinate  units. 


Mean 

Standard  Deviation 

24.  Manner  and  efficiency  with 
which  intelligence  is  gathered, 
reported,  passed  on,  and 
utilized. 


Importance 


Mean 

Standard  Deviation 

25.  Unit  leader’s  demonstration  of 
professional  knowledge  and 
combat  skills. 

Importance 


Mean 

Standard  Deviation 


“"Deleted  from  Round  Tliree 


QUESTION  I RESULTS  (CONT.) 


f 

I 

ROUND 

ITEMS 

TWO 

THREE 

1 26.  Individual’s  demonstration  of 

i combat  and  MOS  skills. 

Imi>ortance 

Mean 

4.70 

Standard  Deviation 

0.50 

m 

27.  Ability  of  unit  to  accomplish 

mission. 

ImiKJrtance 

Mean 

4.80 

4.90 

Standard.  Deviation 

0.40 

0.11 

*28.  MOS  practice  examinations. 

j Importance 

i Mean 

3.00 

! Standard  Deviation 

-t 

1.10 

29.  Ability  to  navigate,  particularly 

at  night. 

Importance 

Mean 

4.50 

4.60 

Standard  Deviation 

0.80 

0.63 

; *30.  Offensive  capability  utilizing 

1 , live  ammunition. 

; lmix)rtance 

Mean 

3.70 

StandardDcviation 

1.20 

; , 31.  Ability  to  carry  out  ambush, 

! i security,  search,  and  attack 

patrols. 

! 1 

1 ! Importance 

1 J Mean 

4.30 

4.00 

j 1 Standard  Deviation 

1 — 

0.70 

0.65 

*Delctcd  from  Round  Three 
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QUESTION  I RESULTS  (CONT.) 


32.  Accountability  and  status  of 
l)crsonnel  and  requests  for 
replacements. 

InuKjrtance 


Mean 

Standard  Deviation 

33.  Physical  condition  of  troops. 
Importance 


Mean 

Standard  Deviation 

34.  Morale,  motivational,  and 
mental  condition  of  troops. 

Importance 


Mean 

Standard  Deviation 

35 . Esprit  and  pride  in/and 
cohesiveness  of  unit. 

Importance 


Mean 

Standard  Deviation 

36.  Appearance  of  troops,  service- 
ability of  uniforms,  boots, 
personal  weapons,  and  size 
of  combat  load. 


Mean 

Standard  Deviation 


ROUND 

TWO 

THREE 

4.10 

3.80 

0.80 

0.47 

4.80 

4.80 

0.40 

0.32 

4.50 

4.40 

0.50 

0.39 

4.70 

4.50 

0.50 

0.42 

4.10 

3.90 

0.70 

0.35 

% UNIT  APPLICABILITY 


ROUND  ROUND 

TWO  THREE 


C 


e C 

C rt.2« 
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QUESTION  I RESULTS  (CONT.) 


ROUND 

ITEMS 

TWO 

THREE 

■>7.  Development  of  plans  and 

orders  - timeliness,  clarity, 
accuracy,  and  adherence  to 
doctrine. 

Importance 

Mean 

4.80 

4.50 

Standard  Deviation 

0.60 

0.67 

38.  Tactical  principles;  followed 

or  ignored? 

Importance 

Mean 

4.70 

4.50 

Standard  Deviation 

0.50 

0.42 

39.  Individuaf  protective  riteasures 

taken  by  troops. 

Importance 

Mean 

4.30 

3.90 

Standard  Deviation 

0.80 

0.53 

*40.  How  well  an/could  personnel 

accept  responsibility  of  a more 
senior  T/0  billet? 

Importance 

Mean 

3.70 

Standard  Deviation 

1.00 

*4 1;.  Supply  economy  and  discipline. 

Importance 

Mean 

3.80 

Standard  Deviation 

1.10 

%UNIT  APPLICABILITY 


ROUND 

TWO 


& c 

= n.2  « 

•a  2 S.-S'o 


<2  o 
* ttOQ 


ROUND 

THREE 
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QUESTION  1 RESULTS  (CONT.) 


ROUND 

THREE 


ROUND 


TWO  THREE 


e C 
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42.  Adequacy  of  reix)rting 
procedures. 

Importance 


Mean 

Standard  Deviation 

43.  Ability  of  a unit  to  read  to 
the  unexpected. 

Importance 


Mean 

Standard  Deviation' 

44.  Ingenuity  and  flexibility  of 
commander  at  all  levels. 

Importance 


Mean 

Standard  Deviation 

45.  Ability  to  execute  team  combat 
functions. 

Importance 

Mean 

Standard  Deviation 

46.  Skill  and  state  of  training  of 
crew-served  weapons. 

Importance 


Mean 

Standard  Deviation 


QUESTION  1 RESULTS  (CONT.) 


47.  Discipline:  immediate  obedience 
to  orders/incidence  of 
misconduct. 

Importance 


Mean 

Standard  Deviation 

*48.  More  innovative  SOPs  to 
improve  professionalism. 

Importance 


Mean 

Standard  Deviation 

**49.  Ability  of  troop  to  operate 
when  burdened  with  special 
equipment,  i.e.  field  protection 
mask,  cold  weather  gear,  etc. 

Importance 


Mean 

Standard  Deviation 

**50.  Ability  of  the  command  to 
function  in  a secure  radio 
mode. 

Importance 


Mean 

Standard  Deviation 


*Deleted  from  Round  Three. 
**Added  in  Round  Two. 
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4.70 

4.20 

0.70 

1.00 

3.30 

1.10 

3.83 

1.24 

4.26 

0.25 

% UNIT  APPLICABILITY 


ROUND 

TWO 


f» 

c = O 4> 

•a  ° S'5'O 

22ela 


ROUND 

THREE 

c|iu 

Ifili 


REVERSE  SIDE  BLANK 


- 4 


APPENDIX  C 

PERFORMANCE  RESPONSE  ITEMS  DEVELOPED  FROM  QUESTION  3 


C-0 


QUESTION  3 RESULTS 


I 

i 


li 

I 

L 

r 


ITEMS 


1.  Ammunition:  (Class  V)  Do 
troops  have  all  they  are  calling 
for? 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 

. Proper  utilization  of  armor. 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 

3.  Effectiveness  of  close  air  support 
in  support  of  attack/defense. 

Iniportance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 

4.  Knowledge  and  use  of  supporting 
arms. 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 


ROUND 

TWO 

THREE 

FOUR 

4.20 

3.60 

3.70 

0.70 

1.10 

1.40 

3.10 

3.60 

I.OO 

1.30 

0:50 

3.80 

3.70 

0.90 

0.80 

3.10 

2.70 

0.90 

0.80 

4.70 

4.60 

0.70 

0.50 

3.10 

3.30 

1.20 

1.30 

5.00 

5.00 

0.00 

0.00 

3.10 

2.60 

1.10 

1.20 

%UNIT  APPLICABILITY 


ROUND 

TWO 


ROUND 

THREE 
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I 


QUESTION  3 RESULTS  (CONT.) 


i 

ROUND 

%UNIT  APPLICABILITY 

ROUND 

TWO 

ROUND 

THREE 

’ 

Squad 

Platoon 

Company 

Battalion 

Brigade 

Squad 

Platoon 

Company 

Battalion 

Brigade 

ITEMS 

TWO 

THREE 

FOUR 

5.  Full  effectiveness  of  supporting 

^ arms  in  all  phases. 

! Importance 

Mean 

4.70 

4.70 

Standard  Deviation 

0.60 

0.50 

1 

1 

1 

Completeness 

1 

1 

1 

1 

1 

Mean 

3.50 

3.10 

1 1 1 
t 1 t 

1 

1 

1 1 1 1 
1 I 1 1 

Standard  Deviation 

1. 00 

1.20 

1 • 1 

• • • 

1 

I i 1 1 1 

6.  Operability  of  combat-essential 

equipment. 

Importance 

Mean 

4.70 

4.80 

Standard -Deviation 

0.50 

0.50 

■ 

1 

1 

i Completeness 

I 

t 

i 1 

Mean 

3.30 

3.10 

1 

1 

1 I 1 1 1 
1 i 1 1 1 

Standard’ Deviation 

0.90 

1.00 

• • 
• • • 

1 

• 

1 i 1 1 1 

7,  Reliability  of  combat-essential 

equipment. 

Importance 

Mean 

4.60 

4.90 

Standard  Deviation 

0.70 

0.40 

1 

1 

1 

j Completeness 

1 

1 

1 

I 1 

Mean 

3.50 

3.00 

1 

1 1 

1 

1 

1 1 1 1 1 
1 1 1 1 1 

StandardiDeviation 

1.10 

i.io 

III 

• » • 

1 

• • 

8.  Execution  of  team  combat 

'■  functions. 

Importance 

Mean 

4 70 

4.90 

' Standard  Deviation 

0.70 

0.40 

1 

1 

1 

1 

1 

I 

Completeness 

1 1 

1 1 
1 1 

Mean 

3.30 

3.00 

i 1 
1 1 

1 1 
1 1 

1 1 1 1 1 
* 1 • 1 1 

Standard  Deviation 

1.00 

1.00 

1 1 

« • • 

• • 

• • • • • 

1 
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QUESTION  3 RESULTS  (CONT.) 


1 


ITEMS 


9.  Execution  of  individual  combat 
skills. 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 

10.  Did  commanders  take  care  of 
men? 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 

1 1 . Ingenuity  and  flexibility  of  com- 
manders at  all  levels. 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 

1 2.  Professional  knowledge  of  com- 
manders and  staff  at  all  levels. 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 


ROUND 


TWO 

THREE 

FOUR 

5.00 

0.00 

2.10 

2.70 

1.00 

1.10 

4;50 

4.60 

0.90 

0.50 

2.90 

2.60 

1.00 

1.50 

4.10 

4.60 

4.70 

1.00 

0.50 

3.10 

3.00 

3.40 

0.90 

1.10 

0.80 

4.80 

5.00 

0.40 

0.00 

3.20 

2.80 

1.30 

1.00 

% UNIT  APPLICABILITY 


ROUND 

TWO 

ROUND 

THREE 

Squad 

Platoon 

Company 

Battalion 

Brigade 

Squad 

Platoon 

Company 

Battalion 

Brigade 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

till 

till 

> ' 


I 

I 
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QUESTION  3 RESULTS  (CONT.) 


ROUND 

ITEMS 

TWO 

THREE 

FOUR 

1 3.  Ability  to  communicate  in  a 

timely  and  accurate  manner. 

Importance 

Mean 

5.00 

4.90 

Standard  Deviation 

0.00 

0.40 

Completeness 

Mean 

2.90 

2.30 

Standard  Deviation 

1.30 

1.30 

14.  Defensive  capability. 

Importance 

Mean 

4.50 

4.90 

Standard  Deviation 

0.70 

0.60 

Completeness 

Mean 

2.50 

2.80 

Standard  Deviation 

0.70 

1.00 

1 5 . End urance  of  personnel  and 

equipment  beyond  the  exercise 
period. 

Importance 

Mean 

3.90 

3:80 

Standard  Deviation 

1.20 

1.20 

Completeness 

Mean 

4.30 

4.70 

Standard  Deviation 

0.80 

0.50 

1 6.  Pliysical,  mental,  and  moral 

fitness. 

Importance 

Mean 

4.30 

4.30 

Standard  Deviation 

1.20 

0.80 

Completeness 

Mean 

3.90 

4.10 

Standard  Deviation 

1.20 

1.10 

%UNIT  APPLICABILITY 
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QUESTION  3 RESULTS  (CONT.) 


r 

► 

I 

; t 


I 


i 


i 

i, 

! '■ 
1 

i , 

I 

i ’ 
i 


ROUND 

ITEMS 

TWO 

THREE 

FOUR 

17.  Unit  performance  in  the  field  for 

an  extended  period. 

Importance 

Mean 

4.50 

4.20 

4.40 

Standard  Deviation 

0.90 

1.20 

0.50 

Completeness 

Mean 

3.70 

4.20 

3.40 

Standard  Deviation 

1.30 

1.10 

0.90 

18.  Execution  in  a combat  situation. 

Importance 

Mean 

4.80 

4.40 

4.60 

Standard  Deviation 

0.60 

1.30 

0.50 

Completeness 

Mean 

3.50 

4.20 

2.80 

Standard  Deviation 

1.40 

1,30 

1.10 

1 9.  Ability  to  deliver  live  fire  accu- 

rately  and  timely. 

Importance 

Mean 

4.90 

5.00 

Standard  Deviation 

0;30 

0.00 

Completeness 

Mean 

3.40 

3.40 

Standard  Deviation 

1.20 

1.10 

20.  Fire  and  maneuver  ability. 

Importance 

Mean 

4.70 

4.90 

Standard  Deviation 

0.70 

0.40 

Completeness 

Mean 

2.90 

2.80 

Standard  Deviation 

1.00 

1.00 

% UNIT  APP 

ROUND 

TWO 


LICABILITY 


ROUND 

THREE 


QUESTION  3 RESULTS  (CONT.) 


ITEMS 


2 1 . Employment  of  organic  weapons. 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 

22.  Fire  discipline. 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 

23.  Definitive  information  on  supply 
and  equipment  status. 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 

24.  Ability  to  sustain  under  certain 
resupply  conditions. 

Importance 

Mean 

Standard  Deviation 

Completeness 

Mean 

Standard  Deviation 


ROUND 

TWO 

THREE 

FOUR 

4.90 

5.00 

0.30 

0.00 

2.80 

2.80 

1.10 

1.00 

4.50 

4.40 

0.70 

0.70 

3.00 

2.80 

1.10 

1.00 

4.20 

4.00 

0.70 

0.80 

2.90 

2.80 

0.90 

1 .20 

4.00 

4.20 

0.80 

0;80 

3.40 

4.00 

0.80 

1.10 

% UNIT  APPLICABILITY 


ROUND 

TWO 


c 

c 0.2  " 

•a  oq.  !='o 

salsa, 

— *2  o 


ROUND 

THREE 


>»  e 

c § 2 « 

o o n 

3 ^ C Wj 

O '■j’C 


QUESTION  3 RESULTS  (CONT.) 


ROUND 

ITEMS 

TWO 

THREE 

FOUR 

25.  Was  supply/resupply  orderly  and 

timely? 

Importance 

Mean 

4.50 

4.30 

Standard  Deviation 

0.70 

Completeness 

Mean 

3.20 

3.30 

Standard  Deviation 

0.70 

1.00 

26.  Supply  economy. 

Importance 

Mean 

4.00 

3.60 

Standard  Deviation 

1.10 

1.40 

Completeness 

Mean 

2.50 

3.40 

Standard  Deviation 

0.80 

1 .30 

27,  Whether  or  not  intelligence  was 

received  and  acted  upon? 

Importance 

Mean 

4.50 

4.00 

Standard  Deviation 

0.70 

0.80 

Completeness 

Mean 

3.30 

2.90 

Standard  Deviation 

1.10 

1.10 

28.  Effectiveness  of  logistic  support 

for  handling  equipment  casualties. 

Importance 

Mean 

4.30 

4.00 

Standard  Deviation 

0.80 

0.80 

Completeness 

Mean 

3.40 

3.40 

Standard  Deviation 

1 

0.70 

1.20 

%UNIT  APPLICABILITY 

ROUND  ROUND 

TWO  THREE 
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QUESTION  3 RESULTS  (CONT.) 


ROUND 

ITEMS 

TWO 

THREE 

FOUR 

29.  Did  mission  get  accomplished? 

Importance 

Mean 

5.00 

5.00 

Standard  Deviation 

0.00 

0.00 

Completeness 

Mean 

2.40 

Standard  Deviation 

1.10 

■ifl 

30.  Patrolling  capability. 

Importance 

Mean 

4.70 

4.50 

Standard  Deviation 

0.50 

0.50 

Completeness 

Mean 

2.80 

2.60 

Standard  Deviation 

0.90 

1.10 

3 1 . Personnel  stability  - are  all 

troops  deployable? 

Importance 

Mean 

3.70 

3.70 

Standard  Deviation 

1.30 

1.40 

Compictetiess 

Mean 

3.50 

4.00 

Standard  Deviation 

1.40 

0.90 

32.  Did  a “frago”  move  the  unit  out 

smartly  after  the  objective  was 
taken? 

Importance 

Mean 

3.60 

4.40 

4.70 

Standard  Deviation 

1.10 

0.50 

0.60 

Completeness 

Mean 

2.30 

2.40 

3.25 

Standard  Deviation 

1.50 

1.50 

1.50 

%UNIT  APPLICABILITY 
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QUESTION  3 RESULTS  (CONT.) 


%UN1T  APPLICABILITY 


ROUND  ROUND 

TWO  THREE 


b C 

MCucaea 


THREE  FOUR 


Did  casualty  reports,  log  reports, 
etc.,  go  up  the  chain  expeditiously, 
and  was  there  rapid  response? 


Importance 


Mean 

Standard  Deviation 


Completeness 


Mean 

Standard  Deviation 


34.  Are  reporting  procedures  faulty 
(S-1 , S-4)  because  of  set  up  of 
problem? 


Importance 


Mean 

Standard  Deviation 


Completeness 


Mean 

Standard  Deviation 


*35.  Do  reporting  procedures  provide 
accurate  information  in  a timely 
manner? 


Importance 


Mean 

Standard  Deviation 


Completeness 


Mean 

Standard -Deviation 


*Tliis  item  was  added  in  Round  Three. 
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APPENDIX  D 


CONTEXTUAL  ITEMS  DEVELOPED  FROM  QUESTIONS 


QUESTION  4 RESULTS 


THREE 


ROUND 

THREE 


c S.S  'J 
•o  o S3 -a 
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1.  Terrain/hydrography. 
Importance 


Mean 

Standard  Deviation 

2.  Weather, 
Importance 


Mean 

Standard  Deviation 

3,  Time  (Available). 
Importance 


Mean 

Standard  Deviation 

4.  Space  (Maneuvering). 
Importance 


Mean 

Standard  Deviation 

5.  Intelligence  (provided  by  higher 
headquarters). 

Importance 

Mean 

Standard  Deviation 

6.  Timeliness  and  quality  of 
requested  close  air  support, 
supporting  arms,  helo  support, 
and  logistics  support. 

Importance 


Mean 

Standard  Deviation 


QUESTION  4 RESULTS  (CONT.) 
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7.  Fatigue  level  of  personnel 
! (due  to  extended,  demanding 

; operation). 

Importance 

Mean 

3.77 

4.33 

Standard  Deviation 

8.  Rules  of  engagement. 
Importance 

1.09 

0.52 

Mean 

3.40  • 

3.88 

Standard  Deviation 

9.  Resources  allocated  by  higher 
headquarters. 

i Importance 

1.34 

0.64 

i Mean 

4.00 

4.29 

} Standard  Deviation 

1 0.  Ovcrcontrol  by  higher 
headquarters. 

Importance 

0.75 

0.49 

Mean 

4.20 

3.63 

Standard  Deviation 

1 11 . Civilians  in  battle  zone. 

Importance 

1.13 

1.19 

' Mean 

3.55 

3.38 

Standard  Deviation 

1 2.  Peacetime  environmental  and 
ft  safety  requirements. 

; Importance 

1.58 

1.06 

i Mean 

3.80 

3.75 

1 Standard  Deviation 

1.58 

0.46 

%UNIT 

APPLICABILITY 
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QUESTION  4 RESULTS  (CONT.) 
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APPENDIX  E 

INITIAL  PERFORMANCE  EVALUATION  FORMAT  FOR 
UNIT  LEVELS  (SQUAD  THROUGH  BRIGADE) 
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NOTE:  hem  numbers  refer  to  the  complete  item  description  found  in  Appendices  B and  D. 


DISTRIBUTION  LIST 


Chid’ of  Naval  Operations  (Op-099) 

(Op-103B) 

(OP-987P10) 

{Op-9840 

Chief  of  Naval  Material  (NMAT  0344) 

(NMAT  035) 

Chief  of  Naval  Personnel  (Pers-lOe) 

Chief  of  Naval  Research  (450)  (4) 

(458) (2) 

Chief  of  Naval  Echioation  and  Training  (N-2) 

(N-5) 

(Code  018) 

Chief  of  Nasal  Technical  Training  (Code  016) 

Chief  of  Naval  Education  and  Training  Support 
Commander  in  Chief.  Pacific  (J30R) 

(RAEC) 

(Science  Advisor) 

Commander  Amphibious  Forces,  Pacific  (Science  Advisor) 

(Code  N-3) 

Commander  Training  Command,  U.  S.  Atlantic  Fleet  (N3A) 

Commandant  of  the  Marine  Corps  (Code  MTMT) 

(Code  MTMT-25) 

(Code  MTMT-5 1 ) 

(RD) 

Commanding  General,  Fleet  Marine  Force,  Pacific  (G-4) 

(R&D) 

Commanding  General,  Landing  Force  Training  Command,  Atlantic 
Commanding  General,  Landing  Force  Training  Command,  Pacific 

Commanding  General,  Marine  Corps  Development  and  Education  Command  (EDCEN  E-03) 
Commanding  Officer,  Fleet  Combat  Direction  Systems  Training  Center,  Pacific  (Code  03A) 
Commanding  Officer,  Naval  Damage  Control  Training  Center 
Commanding  Officer,  Fleet  Training  Center,  San  Diego 

Officer  in  Charge,  Naval  Education  and  Training  Information  Systems  Activity,  Memphis  Detachment 

Naval  AviatioiMntegrated  Logistic  Support  Center 

Naval  Amphibious  School,  Coronado 

Naval  Amphibious  School,  Little  Creek 

Naval  Development  and  Training  Center,  San  Diego  (Code  0120) 

Naval  Communications  Training  Center 
Naval  Training  Equipment  Center 

Naval  Education  and  Training  Program  Development  Center 
Naval  Education  and  Training  Support  Center,  Pacific 
Naval  Academy,  Annapolis 
Naval  Postgraduate  School,  Monterey 

Naval  Research  Laboratory,  Washington,  D.  C.  (Code  2029)  (6) 

Office  of  Naval  d^esearch  Branch  Office,  Pasadena  (2) 

Center  for  Naval  Analyses 

Director  of  Navy  Laboratories  (MAT-03LM) 
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Director  of  Researcli.  U.  S.  Mililarj'  Academy,  West  Point 
Army  Researcli  Institute  for  Behavioral  and  Social  Sciences 
Keesler  Technical  Training  Center 

Occupational  and  Manpower  Research  Division,  Air  Force  Human  Resources  Laboratory  (AFSC). 
Lackland  Air  Force  Base 

Personnel  Research  Division,  Air  Force  Human  Resources  Laboratory  (AFSC),  Lackland  Air  Force  Base 
Technical  Library,  Air  Force  Human  Resources  Laboratory,  Lackland  Air  Force  Base 
l echnical  Training  Division,  Air  Force  Human  Resources  Laboratory,  Lowry  Air  Force  Base 
Flying  Training  Division,  Air  Force  Human  Resources  Laboratory,  Williams  Air  Force  Base,  Arizona 
Advanced  Systems  Division,  Air  Force  Human  Resources  Laboratory,  Wright-Patterson  Air  Force  Base 
Secretary  Treasurer.  U.  S.  Naval  Institute 

Assistant  Director,  Life  Sciences,  Air  Force  Office  of  Scientific  Research 
Interagency  Committee  on  Manpower  Research  (2) 

Defense  Documentation  Center  (1 2) 


20 


